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IN THE UNITED STATES PATENT AND TRADEMARK OFFICE 

In re patent application of 
Boon-Lock YEO et al. 
Filed: Concurrently herewith 

Continuation of: 

Serial No. 08/956,417 Group Art Unit: 271 1 

Filed: October 23, 1997 Examiner: S. Huang 

For: SUMMARY FRAMES IN VIDEO 

Assistant Commissioner for Patents 
Washington, D.C. 20231 

PRELIMINARY AMENDMENT 

Sir: 

Prior to examination on the merits, please amended to above identified 
patent application as follows: 

IN THE SPECTFTCATION: 

On page 1, line 2, after the title, insert the following: 
CROSS REFERENCE TO RELATED APPT JCATTONS 

This is a continuation of U.S. Patent Application Serial No. 

08/956,417, now U.S. Patent , filed on October 23, 1997, the 

contents of which are herein incorporated by reference.— 
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IN THE CLAIMS : 

Please cancel claims 1-20 and substitute therefore the following: 



1 21. A video viewing system, comprising: 

2 means for selecting a programming channel containing video 

3 program in progress; 

4 a display screen for viewing a video program in progress; 

5 at least one summary frame also displayed on said display 

6 screen along with said video program in progress at a same time when 

7 said programming channel is selected, said at least one summary frame 

8 comprising a past frame from said video program in progress. 

1 22. A video viewing system as recited in claim 21 wherein said 

2 selecting said at least one summary frame displays a video segment on 

3 said viewing screen corresponding to said summary frame. 

1 23. A video viewing system as recited in claim 21 comprising a 

2 plurality of said summary frames each corresponding to a past frame 

3 from said video program in progress. 

1 24. A video viewing system as recited in claim 21 further 

2 comprising at least one preview frame comprising a future frame from 

3 said video program in progress. 

1 25. A user interactive video viewing system, comprising: 

2 a display screen for viewing a video program; 

3 at least one summary frame displayed on said display screen at 

4 a same time with said video program, said at least one summary frame 

5 comprising one of a past or future frame from said program; 

6 a control means for allowing a user to change said video 
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7 program and for allowing said user to select said at least one summary 

8 frame to play at least a segment of said video program corresponding 

9 to said selected summary frame. 

1 26. A user interactive video viewing system as recited in claim 25 

2 wherein said viewing system comprises a television system. 

1 27. A user interactive video viewing system as recited in claim 26 

2 wherein said summary frames remain on said display screen when said 

3 video program is preempted. 

1 28. A user interactive video viewing system as recited in claim 26 

2 wherein the user can delete said summary frames from said display 

3 screen. 

1 29. A user interactive video viewing system as recited in claim 25 

2 wherein said viewing system comprises client connected to a server. 

1 30. A method for informing a viewer of the content of a video 

2 program in progress, comprising the steps of: 

3 selecting a plurality of summary frames depicting selected 

4 events from a video program; 

5 embedding said summary frames in said video program; 

6 transmitting said video program comprising said summary 

7 frames over a media; and 

8 displaying said video program and said summary frames on a 

9 screen at a same time with said video program when a viewer selects 
1 0 said video program. 



1 

2 



31. A method for informing a viewer of the content of a video 
program in progress as recited in claim 30, further comprising the 
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3 steps: 

4 displaying a video segment corresponding to a particular 

5 summary frame when said summary frame is selected by a viewer; and 

6 resuming said video program when said video segment has 

7 finished. 

1 32. A method for informing a viewer of the content of a video program 

2 in progress as recited in claim 30, comprising the step of continuing to 

3 display said summary frames when said video program is preempted. 

1 33. A method for informing a viewer of the content of a video 

2 program in progress as recited in claim 30, further comprising the steps 

3 of: 

4 writing selected frames from said selecting step in a row 

5 direction of a table; 

6 reading said selected frames from said table in a column 

7 direction to interleave said summary frames displayed on said screen. 

1 34. A method for informing a viewer of the content of a video 

2 program in progress as recited in claim 30 wherein said step of 

3 displaying said video program comprising said summary frames on a 

4 screen comprises placing said video program and said summary frames 

5 in designated windows on said screen. 



1 

2 
3 



35. A method for informing a viewer of the content of a video 
program in progress as recited in claim 30 wherein said video program 
and said video program in progress are on different channels. 



Y09-97-348B 



Continuation of: Serial No. ^08/956^4 17 

5 



REMARKS 

Claims 1-20 have been canceled. Claims 21-35 are newly presented for 
examination. 

This application is assigned to IBM Corporation as recorded in the 
parent application. 

Briefly, the present invention is directed to a system for placing 
summary frames into a currently playing video depicting either past or future 
scenes so that a viewer can get a quick overview of the video program. As 
explained on page 5 and shown in Figure 1 , the summary frames depict key 
scenes from the past which aid the viewer in quickly ascertaining the current 
plot or theme of the video program. Hence, when a channel surfer arrives at a 
new channel, rather than only having what is currently playing to catch the 
eye, summary frames are also available to catch the surfer's attention and aid 
in understanding the programming and perhaps hold the viewer's attention 
long enough to keep him from immediately changing the channel. 

In the parent application, claims were rejected as being unpatentable 
over either Nakagki or Nakagaki in view of Marshall. 

The Examiner appeared to appreciate that there were differences 
between the claimed invention and the prior art but had concerns that, when 
giving the claims there broadest interpretation, they did not adequately 
distinguish over the prior art. In particular, the examiner noted that it was not 
clear from the claims that the summary frames were available for display 
immediately when the viewer arrives at a particular channel. 

To that end, the claims as now presented are believed to make that 
distinction clear. For example, Independent claim 21 recites "means for 
selecting a programming channel containing video program in progress; a 
display screen for viewing a video program in progress; at least one summary 
frame also displayed on said display screen along with said video program in 
progress at a same time when said programming channel is selected , said at 
least one summary frame comprising a past frame from said video program in 
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progress" (emphasis added). 

Similarly, independent claim 25 recites "at least one summary frame 
displayed on said display screen at a same time with said video program - said 
at least one summary frame comprising one of a past or future frame from said 
program" (emphasis added). 

Likewise, independent claim 30 recites "displaying said video program 
and said summary frames on a screen at a same time with said video program 
when a viewer selects said video program " (emphasis added). 

The above highlighted limitations are not taught or suggested by the 
prior art of record. In particular, the primary reference cited by the Examiner to 
Nakagaki is directed to a television set which includes a delay circuit 
("caching") for recording a television show as it is being watched. This is not 
related to Applicant's claimed invention. 

The Examiner has additionally relied on Marshall for teaching 
"showing future video clips to a viewer on a display screen". However, 
Marshall is directed to a preview channel which lists all of the programming 
currently available and those that will be airing shortly. Most cable and 
satellite systems have such a channel. Included with each listed programming 
item may also be a short preview clip of the program selectable by the viewer, 
thus making Marshall's preview channel user interactive. However, again, this 
has nothing to do with placing summary frames, either past or future, in a 
program broadcast to be displayed when a channel surfer arrives at a particular 
channel. 

It is respectfully submitted that neither Nakagaki nor Hamilton, taken 
alone or in combination, teaches or suggests Applicant's claimed invention 
and that the claims as presented patentably distinguish over this art. 

Examination on the merits, taking the above into consideration is 
respectfully requested at the earliest possible date. 
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Please charge any deficiencies in fees and credit any overpayment of 
fees to Assignee's Deposit Account No. 50-0510 (IBM/Yorktown). 



McGuire Woods LLP 
1750 Tysons Boulevard 
Suite 1800 

McLean, Virginia 22102 
703-712-5000 



Respectfully submitted, 
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SUMMARY FRAMES IN VIDEO 
DESCRIPTION 

5 BACKGROUND OF THE INVENTION 

Field of the Invention 

The present invention generally relates to placing 
10 summary frames in a video and, more particularly, to a 

system for placing summary frames into a currently playing 
video depicting either past or future scenes so that a 
viewer can get a quick overview of the video. 

15 Description of the Related Art 

Television viewers are stymied at the number of viewing 
choices available these days. Cable television systems 
typically offer one-hundred or more channels to viewer 

2 0 subscribers. Digital satellite systems make available 

hundreds of channels for viewing choices. Video signaling 
over, for example, the Internet, allows interactive viewing 
and the downloading of video from virtually millions of 
sites. For better or worse, all of these video and 

25 entertainment choices have made couch potatoes, more kindly 

referred to as "channel surfers", out of many of us. 

In a typical channel surfing scenario, a surfer, armed 
with an infrared remote control, sits or lies across the 
room, carefully aims, and begins repeatedly pressing the 

30 "channel-up" or "channel-down" button in the general 

direction of the television set. A good surfer is oblivious 
to other viewers in the room and the other viewers are 
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usually powerless to stop an experienced surfer- A surfer 
will typically not spend more than two or three seconds on 
any one channel unless there is something on the screen at 
that precise moment which is of interest. Channels with 
5 commercials currently airing and most movies deemed to be 

well in progress are surfed over almost instantly with very 
little pause. Similarly, the two or three seconds that a 
good surfer typically gives a channel before surfing to the 
next may be insufficient time to accurately determine the 

10 true merit of the programing on the channel. 

Based on the above scenario, there is a real need in 
the art for a system to more accurately inform channel 
surfers or other video viewers of the contents of the 
current programming on a particular channel and to offer a 

15 quick overview of the video program to get the viewer 

quickly "up to speed" on programs already in progress, 

SUMMARY OF THE INVENTION 

2 0 It is therefore an object of the present invention to 

provide summary frame windows on the screen of a playing 
video. 

It is yet another object of the present invention to 
allow a viewer to select one of the summary frames to have a 
25 short video clip played on the screen, including the 

accompanying audio, to get the viewer quickly "up to speed" 
on programs already in progress. 

It is yet another object of the present invention to 
allow a viewer to quickly jump back to a past segment of a 
30 video program to aid in understanding what is presently 

happening . 

According to the invention , various shots or summary 
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frames are selected from a video prior to broadcast. These 
summary frames are embedded in the broadcast and appear in a 
small window on the screen at some point in the future as 
the video is broadcast. The summary frames depict key 
5 scenes from the past which aid the viewer in quickly 

ascertaining the current plot or theme of the video program. 
In an interactive television environment, a short video 
segment may be associated with each shot such that when the 
shot is selected, the past video segment is played in a 

10 larger window on the screen complete with sound. Once the 

segment is viewed, the video program is rejoined in progress 
with the viewer having a better sense of what is going on in 
the video. Hence, when a channel surfer arrives at a new 
channel, rather than only having what is currently playing 

15 to catch the eye, summary frames are also available to catch 

the surfer's attention and aid in understanding the 
programming. 

In a television broadcast environment, this invention 
is particularly helpful in the event a preempting commercial 

2 0 is airing when a particular channel is surfed. Since the 

summary frames remain on the screen, the viewer now has a 
way to quickly determine what program is playing and if it 
is worth viewing. Similarly, in the event a program is 
preempted, for example, for an emergency broadcast or an 
25 overtime sports event, summary frames are provided to keep 

the viewer in touch with the regularly scheduled . 
programming. Similar to a television broadcast, the summary 
frames may also be embedded in prerecorded media, such as, 
for example, video tape or video disks. 

3 0 In an interactive video environment (i.e., video 

signaling, for example, over the Internet) , the summary 
frames may allow for quick hyperlinking to a past even 
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future portion of the video • Frames hyper linking to future 
portions are referred to as preview frames. When a hyperlink 
summary or preview frame is selected, the video can jump and 
actually begin playing in its entirety from that point, 

BRIEF DESCRIPTION OF THE DRAWINGS 

The foregoing and other objects , aspects and advantages 
will be better understood from the following detailed 
description of a preferred embodiment of the invention with 
reference to the drawings, in which: 

Figure 1 is a television channel surfing scenario 
according to a first embodiment of the present invention; 

Figure 2 is a is a computer screen for viewing an 
interactive video signal; 

Figure 3 is a computer display showing a main video 
playback window and three summary windows; 

Figure 4 is a block diagram for video processing at a 
server for summary display; 

Figure 5 is a block diagram for video processing at a 
server for summary extraction and display; 

Figures 6A and 6B are candidate summary frames with 
uniform and non-uniform sampling, respectively; 

Figure 7 is a diagram showing candidate summary frames 
with content based sampling; 

Figures 8A-C are examples of candidate summary frames 
selection and display; 

Figures 9A-B is a first example candidate summary frame 
selection/replacement and display, respectively; 

Figures 10A-B is a second example candidate summary 
frame selection/replacement and display, respectively; 

Figures 11A-B is a third example candidate summary 
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frame selection/replacement and display, respectively; 

Figure 12 is a flow diagram for selecting the summary 
frames; 

Figure 13 is a table of interleaved summary frames; and 
5 Figure 14 is a table of second method for interleaving 

the selection of summary frames. 

DETAILED DESCRIPTION OF A PREFERRED 
EMBODIMENT OF THE INVENTION 

10 

As used herein, a "video signal" represents both analog 
and digital video signals. In the case of analog signal, the 
video signal can be displayed by the existing means of 
displaying analog signal (e.g., television (TV)). Processing 

15 an analog signal is accomplished by first digitizing an 

analog video signal to a digital video signal, and 
subsequently processed. 

The invention has application in many areas, such as, 
for example, digital television broadcast, channel-surfing, 

20 video-on-demand, digital video disk systems, interactive 

video playback, Internet video signaling, video databases, 
and digital libraries. Referring now to the drawings, and 
more particularly to Figure 1, there is shown a viewer 10 
channel surfing the television channels according to a first 

25 embodiment of the invention. The viewer 10 is shown sitting 
across the room aiming a remote control unit 12 at the 
television set 14. As the channels are changed, a channel 
identification and audio stereo indicator 16 may be 
displayed on the screen 18 for a few seconds. In this 

30 example, channel 07 is currently broadcasting what appears 

to be a small passenger jet flying through the clouds. This 
video scene may not appear at first glance to be of interest 
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to the viewer 10, Hence, the viewer 10 may be inclined to 
quickly surf to the next channel. 

However, according to the invention, various shots or 
summary frames are selected from the video prior to 
5 broadcast. These summary frames are embedded in the 

broadcast and appear in a small window as shown on the left 
side of the screen 18. In this example, three summary frames 
are shown 20, 22, and 24, respectively. These summary frames 
depict key scenes from the past which aid the viewer in 

10 quickly ascertaining the current plot or theme of the video 

program. Summary frame 20 shows what appears to be a scene 
of a girl playing basketball, summary frame 22 shows a man 
in a suit talking on the phone, and summary frame 24 shows 
an army soldier talking on a portable phone. Based on this 

15 additional information, the viewer 10 has more information 

on which base his decision to surf to the next channel or to 
linger on this channel. A short video segment may be 
associated with each shot such that when the shot is 
selected, for example by aiming and pointing the remote 

2 0 control or entering a key command on the remote control 12, 

the past video segment is played in a larger window on the 
screen complete with audio 28. Once the segment is viewed, 
the video program is rejoined in progress with the viewer 
having a better sense of what is going on in the video. Of 
25 course, the viewer 10 may select to delete the summary 

frames from the screen if they wish. Alternatively, the 
summary screens may automatically disappear from the screen 
if the channel has not been changed for some time. 

Hence, according to the invention, when a viewer 10 

3 0 arrives at a new channel, rather than only having what is 

currently playing to catch the eye, the summary frames, 20, 
22, and 24, are also available to catch the viewer's 
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attention and aid in understanding the programming. This is 
particulary useful in the event a commercial is airing when 
a particular channel is surfed. Since the summary frames 
appearing on the screen allows the viewer to determine what 
5 program is playing and if it is worth waiting for the 

commercial to end. 

Referring now to Figure 2, there is shown a computer 
screen displaying a video window 50. In an interactive video 
environment (i.e., video signaling, for example, over the 

10 Internet) , the summary frames may allow for quick 

hyper linking to a past, and preview frames to a future, 
portion of the video. In this case, when a hyperlink is 
selected, the video can jump and actually begin playing in 
its entirety from that point. For example, as shown in 

15 Figure 3, a boardroom meeting video is being viewed. On the 

lefthand side of the screen, summary screens for three of 
the speakers at the meeting are displayed, 52, 54, and 56. 
Clicking on any of these summary frames will hyperlink to 
that portion in the video and play will begin from that 

20 point. 

Referring now to Figure 3, an alternative method of 
displaying a video window 3 00 together with the summary 
frames 301, 3 02 and 3 03 on a computer screen is shown. Here, 

25 the summary frames 3 01 - 303 are displayed on separate 

windows from the main video window 300. As before, 
interactivity in the form of hyperlinking and playback of 
selected summary frames can be facilitated. 

Referring now to Figure 4, a system of processing video 

3 0 signals for summary display is shown. This is a system when 

the processing of video and embedding of summary is done on 
a server before the video with summary is sent to the 
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clients. We use the clients to refer to the devices such as 
computer or set-top boxes/television which the users employ 
to receive and/or process the incoming video signal. 
At the server 401, video signals 4 03 are first 
5 processed at step 404. At this step, summary frames are 

extracted. At step 405, the summary frames are embedded with 
the video signal 403 to a combined video signal 406. Video 
signal 406 is then either stored back into the server 401 
and transmitted through communication channels 402. The 

10 channels comprise of satellite networks, cables, and other 

wired or wireless means of communications. 

If the video signal 406 is to be transmitted as a 
digital signal (such as in Direct TV broadcast or over the 
Internet) , then the summary frames can also be embedded as a 

15 digital signal into the video signal 406. If the video 

signal 406 is to be transmitted as an analog signal (such as 
in NTBC or PAL broadcast) , then the summary frames can be 
embedded/ transmitted using one of several techniques: (i) 
during the blanking intervals of analog TV signals (ii) 

20 allocation of additional frequency spectrum (iii) using a 

different TV channel (iv) or using separate and different 
channel for transmission, e.g., through telephone lines. The 
video signal containing summary frames (406) can be played 
back on a computer screen 410 of a computer 411. 

25 Alternatively, in a broadcast environment, it is received in 
a set-top box 413, decoded and then displayed on a 
television screen 412. On the television screen 412, the 
summary frames are overlaid onto the main TV program. 

Referring now to Figure 5, a system of processing video 

3 0 signals for summary extraction and display on the client is 

shown. A video signal 503 is transmitted through 
communication channels 502. The signal can either be an 
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analog signal (such as NTSC or PAL) or digital (such as 
DirecTV) . If the client is a computer 511, then processing 
of incoming video signal 504 is performed first at step 506. 
The video signal if in analog form, is first digitized to 
5 digital signal* The summary extracted are then displayed at 

step 507 on a computer display 510. If the client is a 
set-top box 513 and television set 512, then processing 
first takes place at 508* Again, if the incoming video 
signal 505 is analog, it is first digitized. The Summary 
10 frames are then displayed at step 509 on a television screen 

512- If the television display takes as input analog 
signals, then the summary frames are converted back to an 
analog signal and then overlaid on top of the regular TV 
signal for display. The overlay is performed in the set-top 
15 box 513. 

The processes of selecting and displaying summary 
frames is now described. First, summary frames have to be 
selected to represent different video content. A 
straightforward method is shown in Figure 6A. A candidate 
20 summary frame is taken from the video signal at regular time 

interval t. A fixed memory buffer in a computing device can 
be maintained to store the selected frames. The candidate 
summary frames can show the past content or future content, 
if appropriate. In a live-broadcast, summary frames of past 
25 content can be selected. 

In Figure 6B, the frames are selected in a non-uniform 
manner. At a given current frame, it is desired to select 
more frames that are closer in time to the current frame and 
fewer frames that are further in time from the current frame 
3 0 (distant past or distant future) . However, given a fixed 

memory buffer in a computing device, it is difficult to 
maintain such a structure when the current (reference) frame 
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is constantly changing. 

An alternative method to select the candidate summary 
frame is based on content-based sampling of video frames, as 
shown in Figure 7 . A candidate summary frame is selected 
5 only if it is sufficiently different from the previously 

selected summary frame, that is, it is a good representative 
of a contiguous segment of video with similar context* For 
example, 701, 702 and 703 show three candidate summary 
frames. Summary frame 703 is selected because its content is 

10 sufficiently different from 702. 

An example method of displaying the summary frames from 
those selected in Figure 8A is shown in Figure 8B and 8C. 
From the current frame, four frames pi, p2 , p3, and p4 are 
selected as summary frames representing the past, and four 

15 frames fl f f2, f 3 , and f4 are selected as summary frames 

representing the future. In Figure 8B, one method of display 
the summary frames is shown. Here, the summary frames are 
overlaid on the top and bottom of the regular display area. 
The display can be a television display or a computer 

20 display. In Figure 8C, another method is shown wherein the 

summary frames are overlaid in the left and right of the 
display. The number of summary frames and the overlaid 
patterns can be varied. 

In the method of Figure 6 and 7, there tends to be a 

25 lot of repetition in scene structure. The implication is 

that summary frames selected can be similar in content. 
Because the display area is limited and the number of 
summary frames that can be displayed at a time is limited. 
One method to maximize the use of these resources is to 

3 0 maximize the information that the summary content carries. 

Thus, the summary frames will better serve this purpose if 
they show visually differing content, that is, the number of 
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visually similar summary frames in display should be 
minimized. 

Figure 9 to 11 show examples of candidate summary frame 
selection, replacement and display method that achieve this 
5 effect. In Figure 9A, frames 901, 902 and 903 are the 

present set of candidate summary frames, designated as pi, 
p2 , and p3 respectively. In Figure 9B, summary frames are 
displayed in display 900 as shown. 

In Figure 10A, as new frames are observed, a new set of 

10 candidate summary frames are selected. Here, frames 902 and 

903 are replaced because they represent similar content to 
what frames 905 and 906 respectively represent, i.e., 
segment b is similar to segment e, and segment c is similar 
to segment f. The selected frames at this instant are 901, 

15 904, 905 and 906. The summary frames are again designated as 

pi, p2, p3, and p4, and displayed in Figure 10B. In Figure 
11A, the current frame is now at another position. Here, 
frames 904 and 901 are also replaced because newer frames 
that are similar are observed. In this case, segment a is 

20 similar to segment h, and segment d is similar to segment g. 

The current set of summary frames is displayed as shown in 
Figure 11B. 

Figure 12 is a flow diagram showing a preferred 
embodiment for selecting the summary frames (process 404, 

25 506, and 508). As shown in block 30, the set of summary 

frames is denoted as S = {f 3 i/ f S 2/ f 3 3/-*- f 3 L> where the 
relationship Sj<s jn holds. Initially, there are no frames in 
S, i,e,, S={}, At block 32, a video is scanned from the 
first frame for identification of keyframes K n , that is, 

30 initially, the first frame of the video, f t , is selected as 

the keyframe. At block 33, S 3j is removed from S if S s:t is 
older than some time t. This ensures that summary frames 
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displayed on the screen are not too old to be relevant. At 
block 34, the next frame f k is scanned selected as the next 
keyframe K n if the intensity value is greater than the 
previously selected keyframe by some predefined threshold: 
5 i.e., d(f k , K n -i) > 5, at decision block 36. Here, d(,) 

measures the differences in the intensity or other 
properties of the intensity related measurement between two 
frames. Examples include histogram and pixel differences, 
described in Yeo and Liu, "Rapid Scene Analysis from 

10 Compressed Video" , IEEE Transactions on Circuits and Systems 

for Video Technology, Vol.5, No. 6, pp. 533-544, December 
1995, herein incorporated by reference. 

At block 38, after a keyframe K n is identified, it is 
then decided if it should be added to the set of summary 

15 frame S. The decision is made at block 4 0 as follows: K n is 

compared using a difference function d(,) against each frame 
f L e S. The one that yields the smallest difference is 
denoted f m . If d(f m , k n ) < e, i.e., if the keyframe K n and f n 
is sufficiently similar, we replace f m in S by K n and 

2 0 reorder the frames in S by increasing order of frames at 

block 42. If, on the other hand, this relationship does not 
hold, and the set S is not full, K n is inserted into the set 
anyway, as depicted in block 41. This technique is repeated 
until the last frame in the video is scanned, at block 44. 
25 This technique maintains only those summary frames that are 

sufficiently different from each other. Similar frames such 
as different shots of the same talking heads will not be 
maintained as summary frames. Thus, during the presentation, 
more different and diverse images can be displayed. This 

3 0 selection scheme is a one-pass scheme. 

An alternative embodiment is to replace the frames that 
have been in set S by the keyframe K a . This method will 
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maintain in the set of summary frames those frames that are 
similar to each other- Thus, in a sequence of alternating 
shots depicting a conversion between two persons, k few 
frames portraying each of the persons will be maintain as 
5 summary frames. 

Yet another alternative method to embed summary frames 
is to make use of results from a shot detection algorithm. 
An example technique that performs shot detection on MPEG 
(Motion Picture Expert Group) compressed video is described 
10 in Yeo and Liu, supra. A summary frame is then selected to 

represent each shot. This selection scheme is a two-pass 
scheme . 

It is desirable to not only allow the user to be able to 
see summary of video content about some time f prior to the 

15 current playback, but also to allow the user to see the 

summary of the video content about time 2t, 3t, prior to the 
current playback. Referring to Figure 13, to achieve this, a 
preferred method of selecting summary frames from the set S 
= {fsw f S 2/ f S 3f ••• f sL> i s to interleave the selection of 

20 frames. If at each step of embedding, it is desired to 

select and embed some M frames (assuming M divides L) , the 
set of summary frames are listed or written into a table 50 
of width L/M and height M in a row-wise fashion (i.e., place 
the frames in the first row first, from left to right, then 

25 in the second row, and so on), as shown by write arrow 52. 

In this example, L=12, and L/M=3 . At each step of the 
embedding, the frames are read off in a next column, as 
illustrated by read arrow 54. In this manner, we guarantee 
that each set of M summary frames selected represent video 

3 0 content spanning a longer period of time than mere the most 

recen t m summary. An alternative embodiment is to also embed 
the summary of video of the future, i.e. about some time t, 
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2t, 3t, ♦ ♦ . after the current playback. 

Another method of interleaving the selection of summary 
frames is shown in Figure 14. Here, the Summary frames are 
listed row-wise in a triangular structure, i.e., each row 

5 has more columns than the previous one. At each step of the 

embedding, the frames are read off in a column fashion. For 
example, if each embedding step requires 4 frames, then 
fsl, fs2, fs4 and fs7 are first used. At the next step, 
fsl2, fsl9, fs3 and fs5 are used. This provides a 

10 non-uniform sampling in the sense that more summary frames 

are used that are closer to the current frame, and fewer 
frames are used that are further from the current frame. 
This achieves what is presented in Figure 6B. 

In an interactive video playback environment, such as 

15 the desktop or video streaming, the summary frames displayed 

with the playback frames can be further used for 
hyperlinking purposes, i.e., a user could click (or select) 
frames f 31 and the playback will continue from frame f 3i . 
A preferred use of summary-hyper linking is now 

2 0 described. At the playback of some current shot s ie the 

summary frames of the past shots and future I shots are 
simultaneously shown. The user can then easily see and 
replay one of the recent 1 shots or skip to one of the next 
shots; this is in contrast to the current method of video 

25 playback, in which a user can only see the current frame 

being playback. 

Another preferred use of summary-hyperlinking is to 
embed the summary frames not of the current video being 
played back, but to embed the summary of video of another 

30 program. A user can then click (or choose) to jump to that 

particular. An example use would be showing the summary of 
another football game while the video playback is of 
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another, A user can then watch the current game but also 
grasp the summary of the other game. He can also switch to 
the other game directly without the needing to know the 
channel. 

5 For digital video compressed with standards such as 

MPEG-1 and MPEG-2, a preferred method to embed the summary 
frames is at the user-data area, which are specified by a 
particular header. To further ensure that the summary frames 
from the bitstreams can be extracted and displayed from any 

10 point (i.e., not necessarily from the very beginning) in the 

bitstreams, a new set of summary frame is embedded at fixed 
intervals T. For an MPEG-1 video (352 by 240 pixels) at 1.5 
Mb its /see, a DC image takes up 1.8 Kbytes. If in one sec, 4 
DC images are embedded, then the overhead is 7.2/192 or 

15 3.75%. If instead, 1 DC image is embedded in each second, 

the overhead is 0.09375 %. For an MPEG-2 video (704 by 480 
pixels) at 4.0 Mbits/sec, a DC image takes up 7.92 Kbytes. 
The overhead for embedding 4 DC images /sec is 6.1 % and for 
1 DC image/ sec is 1.5 %. For higher bit rate MPEG-2 such as 

20 6.0 M bits /sec, the respective overhead becomes 4 % and 0.38 

%. The embedding process operates directly on the compressed 
video and does not require any decompression, and thus very 
computationally efficient. 

While the invention has been described in terms of 

25 preferred embodiments, those skilled in the art will 

recognize that the invention can be practiced with 
modification within the spirit and scope of the appended 
claims. 
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CLAIMS 



We claim: 

1 1. A video viewing system, comprising; 

2 a display screen for viewing a video program in 

3 progress; 

4 at least one summary frame superimposed on said 

5 display screen along with said video program in 

6 progress, said at least one summary frame comprising a 

7 past frame from said video program in progress. 

1 2 . A video viewing system as recited in claim 1 

2 wherein said selecting said at least one summary frame 

3 displays a video segment on said viewing screen 

4 corresponding to said summary frame . 

1 3. A video viewing system as recited in claim 1 

2 comprising a plurality of said summary frames each 

3 corresponding to a past frame from said video program 

4 in progress, 

1 4, A video viewing system as recited in claim 1 

2 further comprising at least one preview frame 

3 comprising a future frame from said video program in 

4 progress . 

1 5. A video viewing system as recited in claim 1 

2 wherein selecting said at least one summary frame 

3 hyperlinks the video program in progress to begin 

4 playing from said selected summary frame. 
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1 6. A video viewing system as recited in claim 4 

2 wherein selecting one of said summary frame and said 

3 preview frame hyperlinks the video program in progress 

4 to begin playing from said selected one of said summary 

5 frame and said preview frame. 

1 7. A user interactive video viewing system, comprising: 

2 a display screen for viewing a video program; 

3 at least one summary frame superimposed on said 

4 display screen, said at least one summary frame 

5 comprising one of a past or future frame from said 

6 program; 

7 a control means for allowing a user to change said 

8 video program and for allowing said user to select said 

9 at least one summary frame to play at least a segment 

10 of said video program corresponding to said selected 

11 summary frame. 

1 8. A user interactive video viewing system as recited 

2 in claim 7 wherein said viewing system comprises a 

3 television system. 

1 9. A user interactive video viewing system as recited 

2 in claim 8 wherein said summary frames remain on said 

3 display screen when said video program is preempted. 

1 10. A user interactive video viewing system as recited 

2 in claim 8 wherein the user can delete said summary 

3 frames from said display screen. 

1 11. A user interactive video viewing system as recited 

2 in claim 7 wherein said viewing system comprises client 
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3 connected to a server. 

1 12 . A user interactive video viewing system as recited 

2 in claim 11 wherein selecting said at least one said 

3 summary frame hyperlinks said server to start playing 

4 said video from said summary frame. 

1 13. A method for informing a viewer of the content of 

2 a video program in progress, comprising the steps of: 

3 selecting a plurality of summary frames depicting 

4 selected events from a video program; 

5 embedding said summary frames in said video 

6 program; 

7 transmitting said video program comprising said 

8 summary frames over a media; and 

9 displaying said video program comprising said 
10 summary frames on a screen. 

1 14. A method for informing a viewer of the content of 

2 a video program in progress as recited in claim 13, 

3 further comprising the steps: 

4 displaying a video segment corresponding to a 

5 particular summary frame when said summary frame is 

6 selected by a viewer; and 

7 resuming said video program when said video 

8 segment has finished. 

1 15. A method for informing a viewer of the content of 

2 a video program in progress as recited in claim 13, 

3 comprising the steps of: 

4 hyperlinking to a corresponding portion in said 

5 video program when a particular summary frame is 
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6 selected by a viewer; and 

7 playing said video program from said corresponding 

8 portion in said video program. 

1 16. A method for informing a viewer of the content of a 

2 video program in progress as recited in claim 13, 

3 comprising the step of continuing to display said 

4 summary frames when said video program is preempted. 

1 17. A method for informing a viewer of the content of 

2 a video program in progress as recited in claim 13, 

3 wherein said selecting step comprises the steps of: 

4 scanning a first frame of said video and 

5 designating said first frame as a key frame; 

6 scanning a next frame in said video; 

7 comparing said next frame to said keyframe with a 

8 difference function, 

9 if a result of said difference function is 

10 negative scanning another frame in said video, 

11 if said result of said difference function is 

12 true, then designating said next frame as said key 

13 frame; 

14 comparing said key frame with summary frames 

15 currently being displayed to find one with a smallest 

16 d i f f er ence ; and 

17 replacing said one with a smallest .difference 

18 with said key frame. 

1 18. A method for informing a viewer of the content of 

2 a video program in progress as recited in claim 13 , 

3 further comprising the steps of: 

4 writing selected frames from said selecting step 
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5 in a row direction of a table; 

6 reading said selected frames from said table in a 

7 column direction to interleave said summary frames 

8 displayed on said screen. 

1 19. A method for informing a viewer of the content 

2 of a video program in progress as recited in claim 13 

3 wherein said step of displaying said video program 

4 comprising said summary frames on a screen comprises 

5 placing said video program and said summary frames in 

6 designated windows on said screen. 

1 20. A method for informing a viewer of the content 

2 of a video program in progress as recited in claim 13 

3 wherein said video program and said video program in 

4 progress are on different channels. 
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ABSTRACT OF THE DISCLOSURE 

5 SUMMARY FRAMES IN VIDEO 

According to the invention, channel surfers are aided 
by summary frames appearing on the screen along with the 
regular program. These summary frames are embedded in the 

10 broadcast and appear in a small window on the screen at some 

point in the future as the video is broadcast. The summary 
frames depict key scenes from the past which aid the viewer 
in quickly ascertaining the current plot or theme of the 
video program. In an interactive television environment, a 

15 short video segment may be associated with each shot such 

that when the shot is selected, the past video segment is 
played in a larger window on the screen complete with sound. 
Once the segment is viewed, the video program is rejoined in 
progress with the viewer having a better sense of what is 

20 going on in the video. Hence, a when a channel surfer 

arrives at a new channel, rather than only having what is 
currently playing to catch the eye, summary frames are also 
available to catch the surfer's attention and aid in 
understanding the programming. 
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